Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Created new conda module for managing conda packages #40455

Closed
wants to merge 2 commits into from

Conversation

tmoschou
Copy link

SUMMARY

Created new conda module for managing conda packages.

ISSUE TYPE
  • New Module Pull Request
COMPONENT NAME

conda

ANSIBLE VERSION
ansible 2.6.0 (feature/conda-module d104f10b0e) last updated 2018/05/21 01:41:23 (GMT +1050)
  config file = None
  configured module search path = ['~/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = ******/ansible/lib/ansible
  executable location = ******/ansible/bin/ansible
  python version = 3.6.5 (default, Apr 19 2018, 10:56:22) [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)]
ADDITIONAL INFORMATION

From https://conda.io/

Package, dependency and environment management for any language—Python, R, Ruby, Lua, Scala, Java, JavaScript, C/ C++, FORTRAN

Conda is an open source package management system and environment management system that runs on Windows, macOS and Linux. Conda quickly installs, runs and updates packages and their dependencies. Conda easily creates, saves, loads and switches between environments on your local computer. It was created for Python programs, but it can package and distribute software for any language.

Module allows you to install, update or remove multiple packages in one transaction.
It is basically a wrapper around the conda command line utility. The command line utility supports writing output in machine readable json and supports a dry-run mode making implementing an ansible module wrapper relatively simple.

@ansibot ansibot added affects_2.6 This issue/PR affects Ansible v2.6 core_review In order to be merged, this PR must follow the core review workflow. module This issue/PR relates to a module. needs_triage Needs a first human triage before being processed. new_contributor This PR is the first contribution by a new community member. new_module This PR includes a new module. new_plugin This PR includes a new plugin. support:community This issue/PR relates to code supported by the Ansible community. support:core This issue/PR relates to code supported by the Ansible Engineering Team. test This PR relates to tests. labels May 20, 2018
@ansibot
Copy link
Contributor

ansibot commented May 20, 2018

The test ansible-test sanity --test validate-modules [explain] failed with 1 error:

lib/ansible/modules/packaging/language/conda.py:0:0: E307 version_added should be 2.6. Currently 2.5

The test ansible-test sanity --test yamllint [explain] failed with 4 errors:

test/integration/targets/conda/tasks/darwin.yml:14:1: empty-lines too many blank lines (1 > 0)
test/integration/targets/conda/tasks/linux.yml:14:1: empty-lines too many blank lines (1 > 0)
test/integration/targets/conda/tasks/main.yml:4:1: empty-lines too many blank lines (2 > 0)
test/integration/targets/conda/tasks/setup.yml:23:1: empty-lines too many blank lines (1 > 0)

click here for bot help

@ansibot ansibot added needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. and removed core_review In order to be merged, this PR must follow the core review workflow. labels May 20, 2018
@tmoschou
Copy link
Author

ready_for_review

@ansibot ansibot added core_review In order to be merged, this PR must follow the core review workflow. and removed needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. labels May 21, 2018
@jborean93 jborean93 removed the needs_triage Needs a first human triage before being processed. label May 24, 2018
@ansibot
Copy link
Contributor

ansibot commented May 25, 2018

The test ansible-test sanity --test shebang [explain] failed with 1 error:

lib/ansible/modules/source_control/subversion.py:0:0: module should not be executable

click here for bot help

@ansibot ansibot added needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. and removed core_review In order to be merged, this PR must follow the core review workflow. labels May 25, 2018
Added integration tests

Lint errors

Skip freebsd integration tests, enable osx
@tmoschou
Copy link
Author

ready_for_review

@ansibot ansibot added core_review In order to be merged, this PR must follow the core review workflow. and removed needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. labels May 25, 2018
@colin-nolan
Copy link
Contributor

colin-nolan commented May 29, 2018

Have you written this module from scratch or is this code from another project, which has already been tried and tested? You posted a comment on UDST/ansible-conda, indicating you were aware of that codebase but this code seems to be entirely different.

We use UDST/ansible-conda in our production setups but that was only possible after fixing it to work with a lot of oddities of the conda CLI. I would be reluctant to change to this if it becomes core unless it can be shown to pass all the same tests as are in UDST/ansible-conda (there are many more test cases required to demonstrate that the module works than those in this PR). It would also be great to show that this code has also been tried and tested by users in the wild.

As it stands, I do not fully support this PR. However, I do think it would be good for there to be core support for Conda so I endorse the effort.

@tmoschou
Copy link
Author

tmoschou commented May 29, 2018

Hi @colin-nolan,

Firstly happy to add more tests.

This module has been developed internally at my work and now wishing to contribute upstream. We have tried and tested this module in our environments but of course welcome others testing too. I was only aware of the UDST code base after this PR was raised, and noticed that there is an open tick on UDST to contribute upstream, but the PR to was ultimately closed as permission to re-licence/redistribute had not been obtained/verified. And nearly wished to convey to any interested parties watching the issue this attempt was in the works.

Re: testing parity between UDST and this - I note that there are API differences. In particular this module allows installing, updating, removing multiple packages in one transaction. This was particularly important for us as installing packages over a loop, caused problems with dependency resolution where versions of packages were changed multiple times and not to the one intended (and much slower as multiple versions of the same package were downloaded and re-installed). As we pass in a list of packages, we hence don't have a version argument as this is specified using package spec notation (e.g. name: numpy>=1.8,<2|1.9). Contrary to UDST, we deliberately permit state: latest when specifying a version spec, in fact packages with a spec are always treated as if latest for the spec (regardless if state: present).

In regards to your concerns around the oddities of the conda CLI, this version handles various issues that I believe you may be referring to judging by UDST/ansible-conda#9, UDST/ansible-conda#14, etc - in particular a change in json format between 4.2 and 4.3 as well as various regressions between versions that we have encountered. E.g --dry-run mode returning different format, or --quiet being ignored if --json is also specified. This PR's integration suite loops over multiple versions of conda to test these. I have also reached out to the continuum conda discussion forum (@kalefranz) for review/feedback should there be any other oddities, that I may not be unaware of. I could not see any tests in UDST testing for odd behaviours of conda that I am not already testing for. If you have any specific examples please share.

@komailo
Copy link

komailo commented Jun 3, 2018

I will be happy to try this out in our environment and give feedback.

Curious to see if the setup will still run if we rerun the playbook. From what I understood, Ansible will not modify the system if its not required.

@ansibot ansibot added the stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. label Jun 3, 2018
@kalefranz
Copy link
Contributor

Read through the whole PR top to bottom on my computer. On my phone now though, so we’ll see how far this goes. Probably just a first round of comments. These specifically pertaining to the API the module presents.

  • Conda’s core operations are create, install, update, and remove. Just like we don’t need create, I also don’t think we need update. I think the declared state—either present, latest, or absent—fully covers the conda operation type being invoked.

  • Speaking of the present state, conda 4.6 will have a new flag that makes this possible finally. See resolve #6845 add option -S / --satisfied-skip-solve to exit early for satisfied specs conda/conda#7291. The default behavior for conda install is really equivalent to what ansible’s latest is, this case coupled with the conda concept of constraints around a package spec. Ansible’s present is this new -S flag.

  • I strongly encourage changing env to env_name, and if necessary making env an alias. In general I prefer that people think of environments having locations rather than just names. The env key here would just be ambiguous and cause potential confusion.

  • The conda CLI quirks... Help me lock down the behavior, and fix through a deprecation path the things that need to change. If we get a robust set of tests here, I’d be eager to figure out how to add the conda ansible module as a test target on conda’s own CI that runs for each commit.

Ok those are initial drive-by thoughts. I’ll have more I’m sure when I can revisit this back on a proper computer screen.

@tmoschou tmoschou changed the title Created new conda module for managing conda packages [WIP] Created new conda module for managing conda packages Jun 4, 2018
@ansibot ansibot added the WIP This issue/PR is a work in progress. Nevertheless it was shared for getting input from peers. label Jun 4, 2018
@ansibot ansibot removed the stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. label Jun 4, 2018
@ansibot
Copy link
Contributor

ansibot commented Jun 4, 2018

The test ansible-test sanity --test validate-modules [explain] failed with 1 error:

lib/ansible/modules/packaging/language/conda.py:0:0: E307 version_added should be 2.7. Currently 2.6

click here for bot help


- name: get miniconda3 installation script
get_url:
url: "https://repo.continuum.io/miniconda/Miniconda3-{{ conda_bootstrap.version }}-{{ conda_variant }}-x86_64.sh"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/repo.continuum.io/repo.anaconda.com :-)

@aldanor
Copy link

aldanor commented Jun 12, 2018

Wonder if I'm missing something here; how do you install a package into a specific environment with this module? (Also ensuring that environment gets created if it didn't exist in the first place)

E.g., production box has two different conda environments that you want to maintain alongside.

@ansibot ansibot added needs_rebase https://docs.ansible.com/ansible/devel/dev_guide/developing_rebasing.html stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. labels Jun 12, 2018
@ansibot ansibot added the packaging Packaging category label Feb 17, 2019
@zeronewb
Copy link

Sorry to bother you, conda 4.6 released. Any chance of upstreaming this module?

@ansibot ansibot added the botmeta This PR modifies the BOTMETA.yml and this requires special attention! label Feb 27, 2019
@beenje
Copy link

beenje commented Apr 10, 2019

@tmoschou still working on this?
I'd really like to see that PR moving forward. I'd be happy to help if I can.

I did a quick test.

  1. If the environment doesn't exist, I think it should be created. How can you create new environments otherwise?
    - name: install flask
      conda:
        name:
          - flask
          - python=3.7
        state: present
        env_name: myapp
        executable: /opt/conda/bin/conda

fatal: [ics-ans-conda-default]: FAILED! => {"changed": false, "cmd": ["/opt/conda/bin/conda", "list", "--full-name", "--json", "--name", "myapp", "flask"], "msg": "Unexpected format
of command result", "rc": 1, "stderr": "WARNING: The conda.compat module is deprecated and will be removed in a future release.\n", "stderr_lines": ["WARNING: The conda.compat module is
deprecated and will be removed in a future release."], "stdout_json": {"caused_by": "None", "error": "EnvironmentLocationNotFound: Not a conda environment: /opt/conda/envs/myapp", "excep
tion_name": "EnvironmentLocationNotFound", "exception_type": "<class 'conda.exceptions.EnvironmentLocationNotFound'>", "location": "/opt/conda/envs/myapp", "message": "Not a conda environment: /opt/conda/envs/myapp"}}
  1. The --update-dependencies flag doesn't exist in conda 4.6. It should be --update-deps

My opinion is that this module should focus on supporting recent versions of conda (>= 4.5).
conda is not something coming with the OS. It's easy to update.

I can look at the first point if you want.

@bbaassssiiee
Copy link
Contributor

Last Shippable Run was Sep 14th, 2018.

@bbaassssiiee
Copy link
Contributor

please resolve conflicts

@tmoschou
Copy link
Author

Apologies that this PR has been left in limbo. I have changed jobs and roles since opening this PR and the centre that initially supported maintaining this module has wound up. Regrettably I currently do not use Ansible or Conda as much as I used to. I have closed this PR as I feel it best for the community to either fork this module or continue any other efforts that may be in the works.

@bbaassssiiee
Copy link
Contributor

An alternative for this is using this role: https://github.com/dockpack/base_miniconda

@ansible ansible locked and limited conversation to collaborators Aug 26, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
affects_2.6 This issue/PR affects Ansible v2.6 botmeta This PR modifies the BOTMETA.yml and this requires special attention! collection:community.general collection Related to Ansible Collections work core_review In order to be merged, this PR must follow the core review workflow. has_issue module This issue/PR relates to a module. needs_rebase https://docs.ansible.com/ansible/devel/dev_guide/developing_rebasing.html new_contributor This PR is the first contribution by a new community member. new_module This PR includes a new module. new_plugin This PR includes a new plugin. packaging Packaging category stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. support:community This issue/PR relates to code supported by the Ansible community. support:core This issue/PR relates to code supported by the Ansible Engineering Team. test This PR relates to tests. WIP This issue/PR is a work in progress. Nevertheless it was shared for getting input from peers.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet